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Summer; and the 
Field This Fall 


As the new academic year began, 
at least seven American colleges and 
universities were offering courses deal- 
ing with programed instruction; the 
International Cooperation Administra- 
tion and the American Management 
Association were investigating their 
use for very different purposes; school 
systems across the country were ex- 
perimenting with programs and the 
Center was catching its breath after a 
summer packed with conferences, pub- 
lic appearances, and sold-out work- 
shops. [It was a hot summer—for pro- 
graming, too. | 

Some 208 educators attended CPI 
workshops in Chicago, New York, 
Newton, Providence, Tampa, and Ro- 
chester. They included elementary, 
secondary, and college teachers from 
virtually all academic disciplines. De- 
partment heads, reading specialists, 
principals, superintendents, and train- 
ing directors for government and in- 
dustry were among the “students.” 
Taking the group as a whole, the 
average number of years of teaching 
experience was 12.5. 

At the 69th annual convention of 
the American Psychological Associa- 
tion, held in New York over the long 
Labor Day week end, seven sympo- 
sia discussed aspects of programed in- 
struction and seven other meetings 
heard readings of papers. 

Many sessions, although not specific- 
cally listing the topic related directly 
to the field, not the least of which were 
given by the Division of Experimental 
Psychology. Prior to the convention, a 
memorandum was circulated, request- 
ing the establishment of a new APA 
division to deal with educational tech- 
nology, including “teaching machines” 
and programed instruction. Following 
an open meeting, the convention de- 
cided to refer the matter to committee. 

One of the summer’s displays of 
Programing progress came in August 
at a Conference - Exhibit on Pro- 
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gramed Learning and Teaching Ma- 
chines, sponsored by the American 
Management Association in New 
York. On display were devices from 
the simplest of pressed plastic forms to 
the most complex of electronic devices, 
all labeled as teaching machines. The 
programs offered on these machines, 
inter alia, taught salesmanship, opera- 
tion of punch-card devices, the wiring 
of electronic circuits, and the means 
of solving Chinese puzzles. At one 
morning session, debaters argued the 
relative merits of “The Linear Pro- 
graming Approach” and “The Branch- 
ing Programing Approach,” with a 
follow-up discussion on these and other 
programing methods. A similar con- 
ference will be held in Los Angeles 
in November. 

In Washington, the American Coun- 
cil on Education and the I.C.A. joint- 
ly sponsored a four-day conference on 
“New Learning Processes for Devel- 
oping Nations.” The 87 participants 
included leading figures from govern- 
ment, education, industry, and the mil- 
itary. Four of the six policy com- 
mittees at the conference recommended 
exploration of programed instruction 
as a way to hasten the educational 
progress of underdeveloped countries. 

Relieved of daily classroom duties, 
a number of teachers and professors 
all over the country worked to pro- 
gram the subject-matter of their own 
specialties. The National Science Foun- 
dation awarded a grant to the Schooi 
Mathematics Study Group to help the 
group program its imaginative ap- 
pruvach to elementary and secondary 
math teaching; and SMSG held a spe- 
cial meeting in New Haven to help its 
members familiarize themselves with 
the theory and practice of programing. 
At Princeton Theological Seminary, a 
program in Biblical Hebrew was com- 
pleted for the Didak 501 teaching ma- 
chine, under the slogan, “The teach- 
ing of ancient languages goes modern!” 

Research reporting the actual class- 
room use of programed instruction be- 
came available in greater quantity. 


A Denver study of 250 tenth- 
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Inside 
Opinion 


QUESTION: Can a program’s quality be meas- 
ured by how well the students do on existing 
standardized achievement tests? 


ANSWER: Before one uses any standardized 
achievement test for purposes of evaluation, 
it is essential that the content of the test be 
carefully compared with the content of the 
course in conjunction with which it is to be 
used. To the extent that what is measured by 
an existing standardized test of high technical 
excellence corresponds to the content and 
objectives of the course, such a test would 
indeed provide a useful tool in an evaluation 
study. This statement would be true irrespec- 
tive of the medium through which the course 
material is presented, whether it be television, 
self-instructional devices, or teacher-in-class. 


Frederick L. Ferris, Jr. 


Director of Curriculum Studies Educational Testing 
Service, Member of ETS Advisory Committee for the 
Study of Programed Instruction. 


ANSWER: Existing standardized achievement 
tests may provide one- kind of criterion for 
the evaluation of programs. But few, if any, 
of the standardized tests are completely suit- 
able as criterion measures. All too often, large- 
scale achievement tests fail to distinguish 
between the effects of instruction and the 
effects of general ability and motivation. Most 
standardized achievement tests are con- 
structed with an eye to selection procedures 
rather than as measures of instructional 
effects. As a consequence, the test items may 
be confounded and thus not sensitive to spe- 
cific instructional effects. 

Furthermore, the standardized achievement 
tests may present a danger as criteria in that 
they can become a standard for education, 
sometimes fixed in terms of conventions 
rather than in terms of absolute standards 
of excellence. When this happens, the test 
becomes the criterion and the concepts of 
educational mastery are sacrificed to a static 
testing procedure. In view of the new develop- 
ments in curriculum and in view of the subject- 
matter-mastery emphasis of programed in- 
struction, the standardized achievement tests 
should not be considered as adequate criterion 
measures by themselves. 


Robert E. Silverman 


Chairman, Psychology Department, University College of 
Arts and Science, New York University. Author and 
Researcher of progromed instructional materials. 
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THIS FALL (Cont. from page 1) 


grade pupils reported that the Eng- 
lish 2600 program had been particu- 
larly effective, by both standardized 
and non standardized measures, with 
students in a high-ability group. Lake- 
wood, Ohio, reported success with 
the use of the Temac Algebra I ma- 
terials in a summer “algebra-improve- 
ment”’ class. One teacher said that “he 
came to know the students in his class 
better as individuals than in any sum- 
mer school class he had taught before.” 

In Ulster County, New York, 280 
students ranging from 7th to 11th 
grades took tests to determine their 
retention of material taught in a mod- 
ern mathematics program. No signifi- 
cant drop in test sceres was observed 
at 2, 4, 6, 8, or 10 week intervals, but 
a small, significant drop was noted at 
the 12th week. Further tests will be 
given at the six-month mark. 

Colleges offering courses in pro- 
gramed instruction, as of last report, 
include Oregon College of Education, 
Cornell (Industrial Training), Teach- 
ers College, Columbia, Rochester, Sy- 
racuse, Southern California (Seminar 
in Training), and Pittsburgh, where 
a fall workshop will be run by the 
American Institute for Research. Nu- 
merous PTA and other groups will 
hold local study sessions, and confer- 
ences. In mid-November, Purdue Uni- 
versity will host a three-day conference 
on programed instruction. 

Some 40,000 subscriptions to this 
bulletin were entered by the not quite 
150,000 to whom the first issue was 
sent. We are highly appreciative and 
humbly thankful for the affirmative 
feedback. Engaged in school work? 
Not now receiving a copy of this bul- 
letin? Drop a post card with your 
name and address to the Center. 
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INSIDE OPINION (Cont. from page 1) 
QUESTION: Does the introduction of programed 
instruction necessitate a major change in 
classroom administration? 


ANSWER: With respect to organizing classroom 
routine, providing equipment and physical faci- 
lities, and planning the daily schedule, pro- 
gramed instruction will bring no marked 
changes in classrooms where individualized 
teaching is already part of the pattern. 

Excellent precedents suggest ways of han- 
dling the new medium. Individualized teaching 
of reading now takes place in many classrooms. 
Art work is organized on an individual basis. 
Pupils work singly or in small groups on special 
projects. In small high schools, two classes in 
different subjects may be studying in the same 
room under the guidance of one teacher. A 
major concern will be to help the learner re- 
late the outcomes of any programed instruc- 
tion to overall educational goals. 

With regard to facilities, | predict that ex- 
cept for special situations, the classroom will 
not consist of cubicles and machines. There 
may be several teaching machines, one or two 
small projectors, and a variety of programs 
supplementing familiar instructional materials. 

Max R. Brunstetter 


Professor of Education, Teachers College, Columbia Uni- 
versity, Coordinator of Summer Workshop in Programed 
Instruction. 

ANSWER: While programed instruction does 
not necessitate major change within the 
classroom it does make possible — even in- 
vite — such change. The potentials are great 
and highly intriguing. The competent, resource- 
ful teacher, who has resigned in despair at 
the attempt to utilize the best techniques of 
individualized instruction, should find immense 
possibilities through the use of programed 
materials. For example, the elementary al- 
gebra class can be released from its age-old 
lock-step of everyone, all together, ready-or- 
not, moving from point to point. Through the 
use of programed instruction materials, the 
secondary English teacher can reduce the end- 
less, largely unproductive hours expended on 
formal grammar. Programed materials in such 
areaS aS map reading and basic geographical 
concepts might, once more, release precious 
class hours. Resourceful teachers would be 
invited to use the time and the knowledge 
gained in new ways. 

There is nothing inherent in the nature of 
programed instruction which forces teachers 
to new devices or new classroom organization- 
al patterns. Those without a dream, with 
myopic vision, without resources will readily 
reduce it to the status of excellent workbook 
material. There are, however, teachers who 
will find it urging them in the direction of an 
almost endless variety of new ways of organ- 
izing instruction within their classrooms. 

Henry 0. Brunner 
Principal, Memorial Place Junior High School, Manhasset, 


New York, where programmed instructional materials 
were used, Spring, 1961. 











Responses 








The following were made by partici- 
pants in the seven summer workshops 
conducted by the Center: 


“T have been amazed by the variety 
and novelty (interesting) in programs 
when done by teachers. Their pro- 
grams with proper editing seem to be 
better than the commercial ones.” 


Mathematics ‘Teacher, New Jersey 


“The workshop has been the great- 
est stimulation in education that I 
have had in the past ten years. Per- 
haps I should not say the workshop 
per se but the concept of programing 
and its implications for education.” 


English Professor, Oregon 


“The place of a program in the 
program in the classroom has changed 
for me. But I didn’t think of it as a 
tool for teaching but am now quite 
convinced that this is one of its most 
valuable assets.” 


High School English Teacher, Oregon 


“T was sorry we were left to dis- 
cover for ourselves that a certain de- 
gree of creativity can be incorporated 
into good programing. For at least a 
week I felt that the limitations dis- 
couraged any teaching beyond rote 
learning; that there was only one way 
to program.” 


Social Science Teacher, Oregon 


“We would have enjoyed a frank 
discussion regarding a shopper’s view 


on the idea of programing — the most 
valuable experiences were actually in 


constructing programs.” 
Elementary School Teacher, Virginia 





CPI TO GATHER, U. S. O. OF E. TO 
PUBLISH DATA ON PROGRAM USE 


The Center began work late in September or 
an ANNOTATED COMPILATION OF PR‘ 
GRAMED INSTRUCTION USAGE, to be put 
lished under the auspices of the U.S. Offic: 
of Education. Outlines of research will be re 
ported in a form currently being developed 
by a group of educators who have employe 
and tested programs in their school systems 
This cooperative panel consists of: Dr. Henr 
Brickell, Manhasset N.Y., Dr. J. Bernard 
Everett, Newton, Mass., Dr. George Fernan- 
dez, Ulster County, N.Y., Dr. Joseph O. Lore 
tan and Dr. Shelly Umans, New York Cit) 
Schools, Dr. Lloid Jones, Denver, Dr. Ernest 
Weinrich, Huntington, N.Y., Dr. Reginald Neu 
wein, Cleveland. The ‘Patterns of Use’’ ques 
tionnaire will be in the mail to the Bulletin 
mailing list shortly. Anyone wishing to report 
completed or current research studies using 
programed instructional materials is invited 
to request blanks from CPI. An annual sup- 
plement to this compilation will be published 
as part of the Journal of Programed Instru 


tion announced elsewhere in this issue. 
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Innovations, educational or otherwise, are apt to pro- 
voke controversy. Therefore, it should surprise no one 
that a classroom innovation as radi.al as programed in- 
struction and its handmaiden the teaching machine has 
prompted a good deal of controversy during the past 
year. Arguments about programed instruction might be 
grouped into two separate categories. 

Into the first category we may put questions regarding 
the ultimate value and place of programed instruction in 
education. Not only such questions as, “Is programed in- 
struction better than what we now have?” but, “Even 
if it can produce immediate results will its long range 
effects be desirable?’ “Furthermore, how should it be 
used?” “Where does programed instruction fit in a total 
educational program?” As fascinating, valid, and impor- 
tant as such questions may be, they are clearly going to 
be answered, not through debate, but by judicious use of 
and careful experimentation with programs in schools. 

The second category includes such questions as the 
relative merits of the methods of programing now in vogue. 
Here we find long and inevitably fruitless arguments over 
such questions as, “Are multiple choice questions bet- 
ter than constructed response programs?” “Should a 
programer adjust to individual differences through the 
use of branches?” “Should he program from the genera! 
to the particular or vice versa?” This last question has 
prompted the invention of the name “ruleg” (rule and 
then examples) and ‘“‘egrule” (examples and then rule) 
to describe the opposite approaches would be nothing 
more than amusing if the two camps didn’t take them- 
selves quite so seriously. 

“Domestic Squabbles” 

If the questions in the first category are valid, impor- 
tant, and generated by the educator’s desire and need to 
know about programed instruction, many of the ques- 
tions in the second category are at least confusing, if not 
invalid and detrimental to the growth of the field. 

Neither the number nor the intensity of these ‘“‘do- 
mestic squabbles” is likely to decrease in the immediate 
future. It may be valuable to examine some of the char- 
acteristics, causes, and possible effects of this “internal”’ 
controversy on the future of programed instruction. 

Controversy carried on with accurate information 
and motivated by a desire to clarify issues, rather than 
to compete for prestige or for a market, is essential 
to the scientific process. Since programed instruction 
is undeniably an outgrowth of basic research in a be- 
havioral science, one would hope to see a certain 
amount of the rigorous experimental attitude of the sci- 
entist maintained by those who have taken upon them- 
selves the responsibility of translating the results of 
classroom. But a good deal of this rigorous experimental 
attitude has been “lost in translation,” and many of the 
“translators” seem to have forgotten that they once spoke 
a common language. Nor has their “Tower of Babel’ 
been any lofty cooperative enterprise; rather, it has been 
the job of competing for a major market. 

Apparently life in the market place requires this kind of 
bravado. As John Dewey warned in 1929: “To such per- 


PROGRAMED INSTRUCTION - EXPANDABLE OR EXPENDABLE? 


sons science is of value because it puts a stamp of approval 
upon this or that specific procedure. It is very easy for 
science to be regarded as a guarantee that goes with the 
sale of goods rather than as a light to the eyes and a 
lamp to the feet.” Or, as James Holland of Harvard has 
recently stated in Contemporary Psychology: “We (the 
experimental scientists) have promised that research 
would enable an analysis of all prerequisite component 
skills and that we would be able to arrive at a sequence 
of carefully designed and arranged tasks that would de- 
velop these skills; now we seem to be translating these 
skills into procedures for expedient programing 
publishers promise money and fame for converting text- 
books into programs. Only wisecracks about ping-pong: 
playing pigeons are in store for the scientist who contin- 
ues to ignore the teaching machine’s true destiny.” 

No one, including Dr. Holland, can say what the 
“true destiny” of the teaching machine and programed 
instruction will be. No matter how bright a promise the 
future holds, however, it is only going to be dimmed 
by the parochialism of any vested interest, commercial or 
theoretical, which asserts that a particular type of pro- 
graming is best. Such an assertion implies that all things 
can be taught in the same way to all people. 

The Blindmen and The Elephant... . 

Those who would describe the whole of programed in- 
struction solely in terms of their own methodology of 
programing bring to mind the familiar story of the blind 
men and the elephant. Whereas the blind men had no 
means of combining each other’s experience into a realistic 
description of the elephant, however, programers need 
only open their eyes to see the relationship between the 
men and the elephant. The blind men had no way to 
combine each other’s experience into a realistic descrip- 
tion of the elephant, programers need only open their 
eyes to see the relationship between the part of pro- 
gramed instruction they are familiar with and the whole 
they must begin to make room for. Of course no one 
yet knows precisely what the relationship will be be- 
tween the various types of programing that now exist 
but those who consider themselves professionals in this 
field should not allow vested interests or personal bias 
to dictate what this relationship should be. We must 
agree with Dr. Holland and demand that the promise of 
the behavioral scientist be fulfilled. The true profession- 
als in the programing field must devote themselves to 
research to help all programers know what type of pro- 
graming is best suited for teaching particular skills to 
particular students. Then and only then will the field of 
programed instruction be able to fulfill its potential. ‘To 
expect one type of programing to teach all things equally 
well is as unrealistic as expecting one teacher to be able 
to teach all subjects with equal facility. But research in 
programed instruction to date has too often been asked 
to find out what is better than what, rather than what is 
better for what purposes. As Dr. Holland puts it: “We 
find comparisons of programed instruction with conven- 


tional methods which lead their authors to overgenerali- 
Continued on page 6 








CHOOSING A PROGRAM: 
How to Start 


(Ed. Note: The Educational Testing Service, under a grant 
from the Carnegie Corporation, ts planning a study to identify 
valid criteria for the evaluation of programs. 41 joint committee 
of the A.E.R.A., A.P.A., and D.AV JI. is also exploring this 
subject, as reported in the last issue of the Bulletin. This 
article reflects the Center's recommendations pending the estab- 
lishment of validated criteria for programs. The princtpal au- 
thor of this article, Dr. Lassar Gotkin, directs the Center’s field 
testing and implementation studies.) 


Educators interested in the field of programed instruc- 
tion are writing letters to the Center asking for guidance 
in choosing programs. ‘The answers to their questions 
must deal with separate but related problems: 1) the 
evaluation of programs; 2) the kinds of research edu- 
cators can hope to carry out successfully in their schools. 
This article deals with the first; an article in the next 
issue will consider the second. 


1. Is the Coritent Appropriate? 

The title of the program does not necessarily supply 
this information. A training director of a communica- 
tions company found that his students, on the average, 
completed a Basic Electricity program in half the time 
required for conventional instruction. On the test at the 
end, however, students using the programed text scored 
30% below the average score of those taught convention- 
ally. Some of the topics covered in the course, and thus 
in the text, were not covered in the program. Even 
where a publisher provides a thorough breakdown of the 
program's content, the educator must examine the ma- 
terial himself. 

At present some programs are being selected for rea- 
sons other than the acceptability of their content. One 
elementary school’ principal believed enough in the pos- 
sibilities of programed instruction and was so interested 
in learning about the field that he adopted a program 
about whose content he had reservations. As competing 
programs covering similar content become available, edu- 
cators will seek to make their selections on the basis of 
field-test evidence. The “remaining questions considered 
in this article will, therefore, pertain to data evolving 
from the field trials of programs with student groups. 


2. Is the Content Well Programed? 

The technical quality of a program is usually meas- 
ured by analysis of students’ responses reduced to a nu- 
merical “error rate.” For the programer refining and 
revising his product, the analysis of students’ responses 
is essential. The programer gains clues as to his selection 
of step size, of prompts, of ways to introduce new re- 
sponses; generally he discovers whether or not he has 
communicated to the learners. 

But the “error rate” itself is insufficient evidence of 
quality; it is perfectly possible to have low error rates 
and teach little. In the field trial of English 2600, using 
Denver 10th graders, the lowest of three ability groups 








showed an error rate of only 6.5%—but their performance 
on a test covering the material improved, on the average, 
by only two points. Teachers in conventional classes were 
equally unsuccessful with a similar group, but an error- 
rate analysis alone would have been most misleading to 
anyone debating the introduction of this program for 
low-ability students. 


Error rates are affected by the programers’ definition 
of the acceptable responses. A 5th grade program on the 
use of the microscope showed a 70% error rate on part 3 
of the following frame: 


Frame 6. This is a picture of one| 


Notice that it is thicker in the | 
the two 





| and thinner at 


The learners had no trouble filling in and 
but 70% wrote [_ends ] instead of | edges |. 
The students’ failure to give the response the programer 
desired is probably related to the two-dimensional pre- 
sentation of the lens. It would seem, however, that the 
students grasped the concept. If the programer chose to 
accept ends as a correct response, though lenses do not 
have ends, the error rate on the item would be 0%. 









An educator can also be misled by the predominant 
practice of reporting error rates in gross percentages 
across entire programs. A 10% error rate can result from 
very different distributions of errors. Finding that the 
test population errs at the 10% level on all items is far 
different from finding no errors on most items but large 
proportions of errors on a few, especially if these few are 
critical frames. 

Differential error rates may be an appropriate criterion 
in determining whether or not a program is suitable for 
a given group of students. If a class of 8th graders has a 
10% or less error rate on 80% of the items, while a 
comparable group of 7th graders has a 10% or less error 
rate on only 40% of the items, the educator should ques- 
tion the suitability of the program for the younger group 
as a whole. 


Unless considered with other data, error analysis is 
valuable mostly as an indicator to the consumer that the 
program has at least been tested. 


3. What Have the Students Learned From 
The Program? 

Far more important than error analysis are the post- 
test results indicating what the students know upon com- 
pletion of the program. And posttest results alone are 
not enough. With some programs, such as Eigen’s Sets, 
Relations, and Functions, pretest information is usually 
unnecessary because the material is wholly new to the 
student. However, in the less exotic areas of study, stu- 
dents may have considerable knowledge of the subject be- 
fore beginning a program. In a 4th-grade experiment us- 
ing 100 words of a spelling program, 28% of the stu- 
dents knew at least 90 words before starting. The field 
test of a vocabulary building program with 7th graders 
showed considerable evidence not only of previous knowl- 


edge, but of previous knowledge that was incorrect. Post- 
Continued on page 6 
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Faulty Frames 





“Faulty Frames” began in the last issue of this Bul- 
letin with a faulty “frame”! The sequence on the pretix 
anti, as given in the Bulletin, was not the complete se- 
quence presented to students, nor was it an early se- 
quence in the program. I had hoped to indicate this by 
numbering the items in the illustrations as 29, 31, 32, 
34, 35. This very weak “prompt” failed. 





(For those readers who wrote in and were concerned 
about what students would have been led to do had the 
sequence been the whole sequence, I should say that in 
the complete sequence neither the response ‘“Antiarctic” 
nor the response “synpathy” occurred among students 
with adequate reading skills. Students had _ previously 
been taughi when anti- is changed to ant- and when 
prefixes ending in w are spelled with an m. The be- 
havior required of the student at this late point in thé 
program was complex and dependent on iengthy previous 
instruction which the student had in fact received, a 
fact I neglected to mention in my text.) 





Why were the-items in the illustrative sequence not 
consecutive in the real program? If a certain number of 
items is required to get to a point, why not put the 
items in a straight line and get to the point? One of 
the commonest errors of new programers is to do just 
that. 

An incomprehensible item will be spotted immediately, 
if not by the response of your first student, certainly by 
the responses of your first group of students. Unclear 
presentations lead to random errors and a bunching of 
omissions of responses. A faulty analysis of the subject 
matter, as illustrated in the last issue, will appear in a 
piling up of a consistent and unusual error, pointing a 
finger at the programer’s mistake. But a sequence of six 
or eight items teaching a single new response will elicit 
few or no errors. Why, then, since we desire correct 
responding, is such a sequence called a programing mis- 
take? 

For the sake of convenience, the following example of 
the faulty sequence will use the same subject matter as 
the example in the last issue. 

1. An antiGemmunist is a person who works agginst the ideas of 

the Ce ‘ i i inst” is 









3. Akind of germ-ki Ne 
septic. 





4. A remedy that works. 1 tain types of poisons is an 
dote. Pe 
5. A person who behaving is said to 
be 


6. A strong 

is ‘pathy. 
Students will make almost no errors on this sequence. 
lhe rote responding involved here will prevent even the 
“intelligent error” antisympathy. Second, students will 
scan the later items rather than read them carefully. 
Third (a prediction verified by data), when the pro- 
gramer does change the subject, requiring a new response, 





a large number of students will write “anti” in the item 
no matter how absurd this response may be in the new 
context. Students have stopped reading. Fourth, if item 
6 is repeated as a review or test at a later date, a large 
number of students will fail to answer it. They have not 
learned, although they have responded correctly. “Vhey 
have been responding correctly for THE WRONG 
REASON. 

When errors occur, not in the sequence itself, but in 
review items or on the posttest, reasonable people might 
conclude that the students have not been given enough 
practice on the particular response. Programers confront- 
ed with such data tend to lengthen the sequence, provid- 
ing more practice, which compounds the mistake. Even 
when such a repair does lead to competent behavior, it 
is at the cost of efficiency. A more appropriate response 
to such data is to break up the sequence. We want to 
schedule, not multiply, the student’s practice. 


‘ 


A sequence on any subject proceeds from “‘simple’’ to 
complex items. ‘The words the students is constructing 
are increasingly less familiar, the last one in item 6 being 
unfamiliar to most 8th graders, for whom the program 
was written. It would appear that the programer is with- 
drawing the prompt (or cue) provided by familiarity 
with the resulting word. In the last item, the student 
should respond “anti”? because he now knows its defini- 
tion. Item 6 appears to be unprompted, depending com- 
pletely on the student's knowledge of the subject. On the 
contrary, it is more strongly prompted than item 2. Se- 
quence itself is a prompt, and a powerful one. 

The sequence prompt, though its name may be new, 
is not new to teachers. Whether we call the phenomenon 
‘massed practice’ as psychologists do, or “one form of 
cramming” as students do, its results are the same 
momentary brilliance followed by rapid disintegration. 
Like any form of instruction, programed instruction 
must take account of the necessary scheduling of practice 
if the resultant behavior is to be permanently learned. 

The repair of such a sequence depends, of course, on 
the specific subject matter. For the particular subject 
illustrated, the following pattern of insertions allows the 
programer to schedule the “anti” items. (Giving specific 
insertion items would require too much space.) 

1. An anti-Communist is a person who works against the ideas 

of the Communists. The prefix meaning “against” is__ ===. 
2. A person who was “against” the Nazi party in Germany was 
ae 
(Insert sequence: prefix pro- meaning “for,” reviewing pro- mean- 
ing “forward” as in ‘proceed’; contrast pro- and anti- words.) 
10. A kind of germ-killer that works against germs is an 
septic. 
(Insert sequence: when is anti followed by a hyphen, when not.) 
15. A remedy that works against certain types of poisons is an 
dote. 
(Insert prefix a-, contrast with anti-.) 
23. A person who fights against society by misbehaving is said to 
be social. 
(Insert sequence: prefix syn- spelling changes.) 
33. A strong feeling against someone (the opposite of sympathy) is 
cinuhianumes 


Continued on page 6 





FAULTY FRAMES (Continued from page 5) 

The resulting lesson, including work on the roots of 
the words, may reach 35 or more items. The same 35 
items, of course, could be rearranged to take up each 
point by itself, By interlocking the separate develop- 
ments, the programer requires the student to read each 
item, and to retain the information for increasingly 
lengthy periods of time. Many subject matters lend 
themselves to this type of treatment. 

The advice to programers, then, is to break up se- 
quences which require the same or similar responses. 
Lace such sequences with parallel developments of other 
parts of the material. If your review items or posttest 
data lead you to believe that you need to give students 
more and more practice, take a good look at the way you 
have sequenced your items. It may be true that students 
need more practice, but that they need only to have their 
practice better scheduled. If each step forward requires a 
little stretching of the “memory muscle,” permanent re- 
tention will be more efficiently attained. 

SusAN MryYER MARKLE 
Director of Programing, Center for Programed Instruc- 


tion. Author of Programs in English Vocabulary, Critical 
Reading, and Programed Primer on Programing. 


P, IL—E. or E. (Continued from page 3) 

zations. Valid general conclusions can seldom be reached 
through such comparisons because the results depend on 
how good the program is and how good the teacher of 
the conventional method.” This type of polemic “re- 
search” simply leads to further confusion and competitive 
feelings rather than to clarification and a climate of co- 
operation so necessary to scientific progress. 

In conclusion, then, if we are ever going to be able to 
answer questions about the ultimate place of programed 
instruction in education by pointing to its positive and 
wide-ranging effects on our schools, we had better be 
sure that the scope of our method is wide enough to 
make such an effect possible. 

Ideally programed instruction can help the schools of 
this country do better those things which they decide are 
the better things to do. To accomplish this, programed in- 
struction must be both flexible and expandable. However, 
if one-sidedness becomes characteristic of programed in- 
struction, education will justifiably judge programed in- 
struction to be both inflexible and expendable. 

There is no single way to program all subject matter 
for all students just as there is no one best way to teach 
all students at all times. Perhaps the best teacher or pro- 
gramer is the one who knows what way of instruction is 
the better way for some students some of the time. Pro- 
gramed instruction must learn that there are no more 
absolutes in modern education than there are in mod- 
ern science. For, as modern science is confronted with 
an expanding and evolving intelligible universe, modern 
education must deal with an expanding and evolving 
universe of intelligibility. Programed instruction must 
likewise evolve and expand if it is to find a place in the 


educational mainstream. P. Kexnern Kowoext 


President, Center for Programed Instruction, Teacher, 
Ancient History, Collegiate School, New York City. 





CHOOSING A PROGRAM: HOW TO START 
(Continued from page 4) 

test achievement without reference to pretest abilities 
may present a distorted picture. 

Tests used for such evaluation should be made avail- 
able to educators purchasing a program. Silberman has 
pointed out that most criterion tests used for research in 
programed instruction have been short, hastily constructed 
tests. Evans, Glaser, and Homme obtained contradictory 
results for retention using different criterion tests. Test 
results for a certain chemistry unit indicated almost no 
learning until analysis of the program results showed 
that students missed test items requiring application of 
principles to contexts that students’ did not understand. 

4. What are the Characteristics of the 
Student Population(s) Involved in Trial Runs? 

It is not uncommon to find the effectiveness of a par- 
ticular program proclaimed on the basis of startling re- 
sults with one student whose intelligence and previous 
experience with the materials are not revealed. Such sell- 
ing procedure is clearly reprehensible, but we are not 
much better off if we know only that the program was 
tested with a group of students of unspecified ability 
levels. Many programs are being tested in suburban 
schools with students whose mean I. Q. is 115. Since 
posttests have repeatedly produced significant correlations 
with I. Q. and other measures of ability, very different 
results may be expected when a program is used with 
different ability levels. 

Actually, the problems of identifying the appropriate 
intelligence level(s) for programs seem to be more com- 
plicated than those involved in finding the appropriate 
grade level(s). In the Center’s “Research Summary No. 
1,” no differences in learning are reported for 9th, 10th, 
and 11th graders of above-average ability—high-ability 
students learned from the program regardless of grade 
level. Where English 2600 was used on a single grade 
level, the highest ability group, starting from a high pre- 
test score, learned the most; the lowest ability level, which 
had the most to learn, gained little. Educators may find 
it more appropriate to know the student’s I. Q. and 
reading level rather than his grade level in deciding 
whether or not the student can handle a program. 

To learn more about this issue, we plan to test a 
program requiring 8th grade reading comprehension with 
4th through 11th graders. The 4th graders using the 
program are academically talented and on the average 
read at the 8th grade level while most of the 11th grad- 
ers are below grade level in reading comprehension. 

Some publishers seem ready to provide, with their 
programs, data similar to the normative data provided 
for standardized tests. Even when normative data are pro- 
vided, however, educators might wish to check out the 
program for themselves; no school system is exactly like 
any other. Replication of field tests by educators will be 
discussed in the next article. 
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RESEARCH 


Silverman, R. E. & Alter, M. “Re- 
sponse Mode, Pacing, and Motivation- 
al Effects in Teaching Machines.” De- 
partment of Psychology, New York 
University. June, 1961. 


The two experiments reported here were 
among five experiments published by the 
authors as one report. 

I. This experiment compared the 
response and no-response conditions in 
programed text and machine formats. 
Subject Matter: 

A 90-frame program dealing with 
the binary numbers system. The “re- 
sponse” program was a standard Skin- 
ner-type fill-in-the-blank program. The 
“no response’ program consisted of 
one or several complete sentences with 
no blanks. 

Subjects: 

60 college students required to par 
ticipate in psychology experiments. 
Procedure: 

Subjects were screened on the basis 
of a preliminary test and then equal 
numbers of subjects were randomly as- 
signed to machine and programed text 
groups; and to response or no-response 
conditions within each group. After 
completing the programs, subjects were 
given a final test consisting of con- 
structed response questions. 

Results: 

No significant differences either be- 
tween the response and _ no-response 
conditions or between the machine and 
text groups or interaction between the 
variables was found. 

Comment: 

Another study by these authors used 
a basic electricity program. Here too, 
there were no significant differences 
between the response and no-response 
conditions. 

II. This experiment investigated the 
relative effectiveness and the motiva- 
tional properties of three types of pres- 
entation devices: (a.) an elaborate 
electromechanical machine, (b.) a 
crude spool-type machine, and (c.) a 
programed text. 

Subject Matter: 

Two 90-frame Skinner-type pro- 

grams: the binary number program 
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cited above, and a portion of the Hol- 
land-Skinner program in psychology. 
Subjects: 


60 paid volunteers: 30 high school 
students and 30 college students. 


Procedure: 

Subjects were not familiar with the 
material. T’en high school and ten col- 
lege students were randomly assigned 
to each of the three training devices. 
Each subject read through one of the 
programs and was then recalled, after 
an average interval of one week, to 
read through the other prograrr. A 
constructed response test was admin- 
istered at the end of each program. 


Results: 

Analysis of the final test scores in- 
dicate that: 

(1) the three devices were equally 
effective, and, (2) the subjects per- 
formed as well in session 2 as they 
had in session 1 on each of the three 
devices. Acclimation to the devices did 
not seem to affect the subjects’ per- 
formance significantly, (3) college stu- 
dents performed significantly _ better 
than high school students and, (4) 
none of the variables interacted signifi- 
cantly with any of the others. 


Comment: 

It would appear, again, that the type 
of device used to present the program 
is not important. As the authors point 
out, however, there is some question 
as to whether two exposures is sufh- 
cient to allow novelty effect to wear 
off. In evaluating the motivational 
properties of “teaching machines’, it 
would probably have been more de- 
sirable to give the subjects more than 
two exposures, 


General Comment: 

The research might have presented 
more information; e.g., time required 
to complete program, IQ, program er- 
ror rate, etc. 

The programs used are brief (87 to 
90 frames), and the sampling pro- 
cedure virtually guarantees subjects of 
relatively high motivation. For highly 
motivated students in a novel experi- 
mental situation all the conditions may 


be equally effective even though pro- 
longed exposure in classroom sur- 
rofindings, with wide variations in 
motivation, might show an advantage 
for one or another means of presenta- 
tion. For these reasons, the authors 
are conservative in making recom- 
mendations based on these findings. 
Criticisms of this nature are applic- 
able to many, if not most, research 
studies involving programed instruc- 
tion; the failure to find significant dif- 
ferences does not necessarily prove the 
absence of such differences. 
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Roe, V. k., Case, H. W., and Roe, A.., 
“Scrambled vs. Ordered Sequence in 
Auto-Instructional Programs.” Teach- 
ing Systems Research Project, Depart- 
ment of Engineering, UCLA, 1961. 


This study investigated the impor- 
tance of sequence in a program. 
Program: 

Mathematics. 71 frames were taken 
from the program on elementary prob- 
ability developed at the UCLA De 
partment of Engineering. 

Subjects: 

36 freshman psychology students 
classified into upper, middle, and low- 
er ability groups. 

Procedures: 

Subjects were divided at random 
within each group, half taking the 
ordered sequence and half the scram- 
bled sequence. The design, is 3 x 2, 
with six subjects in each cell. 

Each item was presented on a 4 x 6 
card, together with a multiple-choice 
statement, Students responded by re- 
moving a patch of rubber-cement be. 
low the answer they thought was cor- 
rect. After completing the program, 
each student was given an _ attitude 
questionnaire and a criterion test. 
‘Time taken to complete the program 
and time taken to complete the test 
were recorded. 


Results: 

Students in the upper ability groups 
showed lower error rates on the pro- 
gram and higher test scores than stu- 
dents in the lower ability groups. 
Both differences were significant. 
Within each group, however, there 
was no significant difference—in time 
required for learning error rate, or 
test score—between those who worked 


Continued on page 8 








RECENT RESEARCH 
(Continued from page 7) 


on the ordered sequence and those who 
worked on the scrambled sequence. 


Comment: 

In a previous study, a student 
who had tailed to read the instruc- 
tions in his programed text had worked 
through the program by reading down 
rather than by moving 
divisions from 


each page 
along the horizontal 
page to page. This student, neverthe- 
less, got a high score on the criterion 
test. The researchers may have felt 
like Hugh Walpole’s princes of Ser- 
endip, who ventured forth on a jour- 
ney and made marvelous discoveries of 
things they were not seeking. 

The authors’ conclude that col- 
lege level students may not require 
such careful sequencing of auto-in- 
structional previously 
been supposed.” The authors further 
state, “It would be interesting to ex- 
amine the scrambled vs. ordered se- 
quence question with students of var- 
ious age levels with programs of 
different length, and in different sub- 
ject areas, and with retention tests 
made at spaced intervals of time.” 
Agreed. 
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Gagne, R. M. and Dick, W., “Learning 
Measures in a Self-Instructional Pro- 
gram in Solving Equations.” Depart- 
ment of Psychology. Princeton Uni- 
versity, 1960. 


This study investigated three di- 
mensions of “what was learned” from 
a beginning algebra problem. 
Program: 

Mathematics. The program ‘“Solv- 
ing Equations,” assumes knowledge of 
6th grade arithmetic. The experiments 
identified 21 equation-solving concepts 
in the program. ‘The desired terminal 
behavior was the solution of simple 
8 booklets. 245 


linear equations. 


frames. 
Subjects: 

Fifty-two 7th grade students of 
“middle-ability” in mathematics. 


Procedures: 

Students completed one booklet a 
day for 8 days. 20 to 50 minutes were 
required to finish lessons. ‘wo intern- 
al variables—error rate and time re- 
quired to complete the program— 
were measured for each subject. Three 


tests of learning—(1) Verbalization 
of concepts, (2) Performance in using 
the concepts, and (3) solution of un- 
familiar equations—were administered 
immediately following completion of 
the program. Six weeks later, verbal- 


ization and performance tests were 
given again. No delayed ‘Transfer 
‘Test was administered. 

Results: 

(1) Internal variables: error rate 
was 6.7%. Mean learning time was 267 
minutes with the range 190 to 380. 
‘The standard deviation was 37 min- 
utes. 

(2) External variables: Verbaliza- 
tion lest mean was 14.0 (of 21) on 
the immediate and 13.0 on the 6-week 
delayed t ‘Test mean 
was 10.5 ie immediate 
and 9.5 on _k-delayed test. 
‘These results indicate considerable re- 
tention, though the loss in both in- 
stances is __ statistically significant. 
Transfer Test: mean performance was 
2.1 out of a possible 50. This result 
indicates little ability to solve new 
equations. 

(3) Relationships among measures: 

(a) Correlations between Transfer 
and other criterion tests ran from .29 
to .44. 

(b) the range of correct responses 
on the Performance Test ran from 
14% to 98%. The 8 test items show- 
ing the lowest level of achievement 
(each item was missed by more than 
50% of the students) involved the 
more complex procedures “taught” in 
the program. Only 14% of the sub- 
jects developed the terminal behavior 
of solving simple equations. 
Comments: 


Particularly valuable is the authors’ 
introductory discussion of internal and 
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external variables as criteria for evalu- 
ating programs. 

Results indicate that “what was 
learned” was retained but not trans- 
ferred to solving ‘new’ equations, 
though the new -equations were “not 
considered particularly difficult by the 
experimenter.” 

Given the low achievement on the 
procedural concepts of the Perform- 
ance Test (only 14% could solve sim- 
ple equations), the low transfer scores 
are not surprising. The authors state, 
“Regardless of the internal criterion 
measures . . . (the error rate was 
6.7%) . . . the transfer test scores 
make one reluctant to state that the 
learning program has truly taught 
“equation solving.” ‘This study dem- 
onstrates the limitations of error rate 
as a criterion for assessing the effec- 
tiveness of programs and the impor- 
tance of using meaningful tests of 
terminal behavior. 
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ium for all those who ar involved in pro 
graming research, and others interested 
research developments in this field. Theor 
etical articles on programing will also be in 
cluded from time to time. Rheem Califone 
feels that it is appropriate that this service be 
carried on by a non-profit organization. 

PROGRAMED INSTRUCTION, the Center 
information bulletin will maintain its present 
combination of news and articles on program- 


ing, plus research abstracts 
The editor of will be Lincoln F. Hanson 
an experimental psychologist of the CPT staff 


The protem editorial board will include Drs 
John Barlow, Bernard Everett James Hol 
land, David Klaus, Douglas Porter and Leon 
ird Silvern. Articles for the Journal houlkk 
be directed to Dr. Hanson, c/o The Center. 


A free initial distribution of the JPI wil 
be made late this fall to the Rheem mailing 
list and others requesting it promptly. Subse 
quent issue will be available only a subscrip 
tion basis. Researchers are invited to submit 
reports to JPI: such reports will be published 
as soon as possible after acceptance. 
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